Restore commented out acceptance test scenario by simonbaird · Pull Request #3505 · conforma/cli

simonbaird · 2026-08-18T19:56:36Z

I believe it was failing at some point, so it was commented out. https://redhat.atlassian.net/browse/EC-1023 is meant to describe the reason for the failure.

It's not clear to me what the detailed story is, but this comment and the revert PR it is part of are definitely clues: #2176 (comment)

Seems to work fine now, so let's bring it back. See also 4461812 to help understand the purpose of the test.

I believe it was failing at some point, so it was commented out. https://redhat.atlassian.net/browse/EC-1023 is meant to describe the reason for the failure. It's not clear to me what the detailed story is, but this comment and the revert PR it is part of are definitely clues: conforma#2176 (comment) Seems to work fine now, so let's bring it back. See also conforma@4461812 to help understand the purpose of the test.

coderabbitai · 2026-08-18T19:56:59Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: 35dff9b5-9b3a-438c-b48e-299c86c73293

📥 Commits

Reviewing files that changed from the base of the PR and between 09cafb4 and 2d699e2.

📒 Files selected for processing (1)

features/validate_image.feature

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

📝 Walkthrough

Walkthrough

The pull request re-enables an acceptance scenario for validating a signed snapshot with ten components and ten policy sources. The scenario checks successful validation and matching JSON output.

Changes

Snapshot validation

Layer / File(s)	Summary
Many components and sources acceptance test `features/validate_image.feature`	The active scenario configures ten policy sources with distinct `ruleData` patches, creates a signed ten-component snapshot, validates it, and checks exit status 0 with matching JSON output.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 2d699

This change restores a previously commented-out acceptance-test scenario without introducing a merge-blocking product or production risk; it is merge-ready after normal checks and review.

Suggested reviewers: cuipinghuo, robnester-rh

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly states that the commented-out acceptance test scenario is restored.
Description check	✅ Passed	The description explains the change, provides context for the prior failure, and links the related ticket and historical references.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

qodo-for-conforma · 2026-08-18T19:57:25Z

PR Summary by Qodo

Restore multi-source image validation acceptance scenario

🧪 Tests 🕐 Less than 5 minutes

AI Description

• Re-enables acceptance coverage for validation with ten components and policy sources.
• Verifies successful execution and stable JSON output against the existing snapshot.

Diagram

graph TD
  S["Acceptance scenario"] --> K["Known key"] --> R["Policy repository"] --> P["Ten policy sources"] --> V["Image validation"] --> A["Status and snapshot"]
  S --> C["Ten components"] --> V

High-Level Assessment

Re-enabling the existing scenario is the most direct approach because its fixtures, command invocation, and expected snapshot already exist. Rewriting or splitting it would add churn without improving coverage for the restored multi-component, multi-source behavior.

Files changed (1) +19 / -20

Tests (1) +19 / -20

validate_image.featureRe-enable many-components-and-sources acceptance coverage +19/-20

Re-enable many-components-and-sources acceptance coverage

• Uncomments the acceptance scenario that validates ten signed components against ten policy sources with distinct rule data. The scenario asserts successful command completion and JSON output matching the existing snapshot.

features/validate_image.feature

fullsend-ai-review · 2026-08-18T19:57:48Z

🤖 Finished Review · ✅ Success · Started 7:57 PM UTC · Completed 8:08 PM UTC

Commit: 87c4a29 · View workflow run →

qodo-for-conforma · 2026-08-18T19:58:23Z

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Tip of the day

💡 Did you know, you can keep summaries lean with Finding overflow, which tucks the rest behind 'View more'

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

fullsend-ai-review · 2026-08-18T20:08:26Z

Looks good to me

codecov · 2026-08-18T20:12:36Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag	Coverage Δ
acceptance	`54.42% <ø> (+0.04%)`	⬆️
generative	`12.28% <ø> (ø)`
integration	`23.59% <ø> (ø)`
unit	`72.20% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

st3penta

LGTM. Thanks, i'll revert the snapshot cleanup in #3489

fullsend-ai-retro · 2026-08-19T09:31:30Z

🤖 Finished Retro · ✅ Success · Started 9:31 AM UTC · Completed 9:41 AM UTC

Commit: 87c4a29 · View workflow run →

fullsend-ai-retro · 2026-08-19T09:41:48Z

Retro: PR #3505 — Restore commented out acceptance test scenario

This was a straightforward, well-functioning workflow. PR #3505 was a human-authored, single-file change by simonbaird that uncommented a previously disabled Gherkin acceptance test scenario in features/validate_image.feature (19 additions, 20 deletions). The review agent ran once, correctly identified the change as trivial/mechanical, and approved it. Human reviewer st3penta also approved with a useful follow-up note. The reviews were fully aligned — no false positives, no unnecessary change requests.

Observations (no new proposals — all covered by existing issues)

1. Sonnet model failure caused ~2.5 min delay. The review orchestrator dispatched two sub-agents (style-conventions, intent-coherence) targeting a Sonnet-tier model on Vertex, but the model was unavailable. After the failure, they were redispatched on Opus, adding ~2.5 minutes of wasted wall time. This is additional evidence for fullsend-ai/fullsend#1771 (model fallback for unavailable models), fullsend-ai/agents#379 (sub-agent model fallback), and fullsend-ai/fullsend#423 (ensure all required models are enabled in GCP).

2. Full sub-agent pipeline ran on a trivial uncomment PR (~12 min, all Opus). The review agent classified the PR as "small" but still dispatched 3 dimension sub-agents + 1 challenger, all ending up on Opus after the Sonnet fallback. For a 39-line change that just removes comment markers from existing test code, a lightweight fast-path would have been appropriate. This is additional evidence for fullsend-ai/agents#301 (lightweight path for trivial/mechanical changes) and fullsend-ai/fullsend#5777 (model routing based on diff complexity, not just author type).

3. Challenger sub-agent worked well. The adversarial reviewer correctly downgraded a borderline indentation finding from low/actionable to info/not-actionable, preventing a false-positive comment. This is the quality mechanism working as designed.

Autonomy assessment

The review agent and human reviewer reached identical conclusions (approve, no comments on code). For this class of change — re-enabling existing commented-out test code with no logical modifications — the review agent demonstrated appropriate judgment. However, this is a minimal-risk scenario that doesn't provide strong signal about autonomy for more complex changes.

github-actions Bot added the size: S label Aug 18, 2026

simonbaird mentioned this pull request Aug 18, 2026

Use ValidateVSAAndComparePolicy for ec validate image VSA skip #3489

Merged

fullsend-ai-review Bot approved these changes Aug 18, 2026

View reviewed changes

fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Aug 18, 2026

st3penta approved these changes Aug 19, 2026

View reviewed changes

st3penta merged commit 7d116df into conforma:main Aug 19, 2026
23 checks passed

simonbaird deleted the restore-commented-out-test branch August 19, 2026 13:53

fullsend-ai-retro Bot mentioned this pull request Aug 19, 2026

Review agent should check for remaining codebase references when test data is deleted fullsend-ai/agents#884

Open

Navigation Menu

Uh oh!

Restore commented out acceptance test scenario - #3505

Restore commented out acceptance test scenario#3505
st3penta merged 1 commit into
conforma:mainfrom
simonbaird:restore-commented-out-test

simonbaird commented Aug 18, 2026

Uh oh!

coderabbitai Bot commented Aug 18, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

qodo-for-conforma Bot commented Aug 18, 2026

Uh oh!

fullsend-ai-review Bot commented Aug 18, 2026 •

edited

Loading

Uh oh!

qodo-for-conforma Bot commented Aug 18, 2026

Uh oh!

fullsend-ai-review Bot commented Aug 18, 2026

Uh oh!

codecov Bot commented Aug 18, 2026

Uh oh!

st3penta left a comment

Uh oh!

Uh oh!

fullsend-ai-retro Bot commented Aug 19, 2026 •

edited

Loading

Uh oh!

fullsend-ai-retro Bot commented Aug 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

simonbaird commented Aug 18, 2026

Uh oh!

coderabbitai Bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

qodo-for-conforma Bot commented Aug 18, 2026

PR Summary by Qodo

Uh oh!

fullsend-ai-review Bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qodo-for-conforma Bot commented Aug 18, 2026

Code Review by Qodo

Great, no issues found!

Uh oh!

fullsend-ai-review Bot commented Aug 18, 2026

Uh oh!

codecov Bot commented Aug 18, 2026

Codecov Report

Uh oh!

st3penta left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fullsend-ai-retro Bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-retro Bot commented Aug 19, 2026

Retro: PR #3505 — Restore commented out acceptance test scenario

Observations (no new proposals — all covered by existing issues)

Autonomy assessment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderabbitai Bot commented Aug 18, 2026 •

edited

Loading

fullsend-ai-review Bot commented Aug 18, 2026 •

edited

Loading

fullsend-ai-retro Bot commented Aug 19, 2026 •

edited

Loading